Papers with diagnostic analysis
WikiTalkEdit: A Dataset for modeling Editors’ behaviors on Wikipedia (2021.naacl-main)
Copied to clipboard
| Challenge: | Using the WikiTalkEdit dataset, we show how positive emotion and the use of first-person pronouns predict a positive emotional change in a Wikipedia contributor. |
| Approach: | They introduce and analyze WikiTalkEdit, a dataset of conversations and edit histories from Wikipedia, for research in online cooperation and conversation modeling. |
| Outcome: | The proposed dataset supports the classic understanding of style matching, where positive emotion and the use of first-person pronouns predict a positive emotional change in a Wikipedia contributor. |
ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models (2024.emnlp-main)
Copied to clipboard
Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana
| Challenge: | Currently, tool-augmented large language models (LLMs) only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100. |
| Approach: | They propose a multi-level diagnostic process to assess the LLM's hallucinations through two perspectives: depth and breadth. |
| Outcome: | The proposed diagnostic process assesses the hallucinations of large language models through two perspectives: depth and breadth. |
Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) achieve strong performance on metaphor detection and interpretation tasks, yet it remains unclear what such success actually reveals about metaphor processing. |
| Approach: | They propose to probing semantic attribute alignment, lexical invariance, and syntactic sensitivity to examine the limits of behavioral evidence for metaphor processing. |
| Outcome: | The proposed model can exhibit semantic drift relative to reference attributes, stable lexical anchors persist across contextual conditions, potentially supporting conventional metaphors while biasing novel metaphors requiring contextual integration. |